The influence of the nonlinear propagation effect on three 400 Gb/s/ch (400G) optical fiber communication systems with typical modulation formats, dual-carrier 16-quadrature amplitude modulation (16QAM), single-carrier 16QAM (single-16QAM), and four-carrier quadrature phase-shift keying, are investigated. The received optical signal-to-noise ratio (OSNR), affected by the nonlinear interference noise together with the amplified spontaneous emission noise, are compared with three 400G systems and a standard 100 Gb/s/ch system by numerical simulations. Both single channel and multichannel cases are considered. Single-16QAM is found to have the best OSNR among those modulation formats.